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PERSONNEL  AND  COLLABORATORS 

The  project  started  on  the  1st  of  November  2005.  The  PhD  student  employed  on  the  grant  was  Vadim  Tikhanoff.  Professor 
Angelo  Cangelosi,  Principal  Investigator  (PI),  is  also  the  student’s  first  supervisor  (Director  of  studies).  Dr.  Tony  Belpaeme, 
from  Plymouth  University,  is  the  second  supervisor  of  the  PhD  student. 

From  the  beginning  of  the  grant  we  have  closely  collaborated  with  Dr.  Leonid  Perlovsky,  who  was  a  named  collaborator  in  the 
proposal.  In  addition,  we  are  collaborating  with  Prof.  Fontanari,  who  holds  another  EOARD  grant  on  a  similar  topic.  This  has 
been  a  very  successful  and  productive  collaboration  and  is  reflected  in  the  publication  of  a  joint  publication. 


OBJECTIVES  AND  ACTIVITIES 

The  tables  below  provide  an  overview  of  the  original  objectives  and  the  list  of  models  and  simulation  experiments  carried  out 
during  the  project.  More  details  on  the  actual  simulation  results  are  described  in  the  enclosed  report,  in  the  publications,  and  in 
the  project  website.  Note  that  text  in  bold  refers  to  the  work  produced  between  April  and  October  2006  (i.e.  after 
submission  of  interim  grant  report).  A  detailed  description  of  the  new  simulation  setup  and  results  is  enclosed  at  the 
end  of  this  document. 


Original  Objectives 

Activity 

1.  To  adapt  the  MFT  algorithm  for  the 
acquisition  of  linguistic,  cognitive  and 
sensorimotor  abilities  in  adaptive  agents 

1.1  The  original  MFT  algorithm  has  been  extended:  (a)  to  deal  with  multi¬ 
feature  representation  of  objects  (see  simulations  in  3.2)  (b)  to  model  the 
acquisition  of  words  (see  simulations  in  3.3);  (c)  to  scale  up  lexicon 
and  model  the  combination  of  words  to  express  complex/composite 
actions  (see  new  simulations  in  4.4-4. 6) 

2.  To  design  and  implement  a  computer 
simulator  for  linguistic  agents  with  MFT 
controller 

2.1  Tikhanoff  has  extended  the  original  cognitive  robotics  simulator, 
originally  developed  by  Hourdakis  &  Cangelosi,  to  deal  with  multiple 
objects  and  actions.  This  has  been  developed  in  C++  with  the  use  of 

ODE  libraries  for  physics  routines  and  OpenGL  for  graphics. 

2.2  Simulations  on  the  MFT  algorithm  were  first  developed  in  Matlab,  and 
then  implemented  in  C++.  This  permitted  the  easy  integration  with  the 
robotics  simulator. 

2.3  The  robotic  and  MFT  simulator  has  been  extended  further  to 
include  experiments  on  multi-stage  learning  of  action  and  words 

This  for  example  permits  the  acquisition  of  basic  actions,  and  their 
name,  and  the  subsequent  combination  to  produce  complex  actions, 
(see  simulation  experiments  in  4.4-4. 6) 

2.4  During  the  Robotics  Summer  School  (July  2006)  the  multi¬ 
dimensional  MFT  algorithm  was  modified  and  tested  on  a  23  degree 
of  freedom  humanoid  robot.  This  successfully  produced  the 
categorization  of  actions  of  the  real  robot. 

3.  To  execute  a  series  of  simulation  experiments 
on  the  integration  of  language  and  cognitive 
abilities  in  collaborative  tasks  (e.g.  object 
manipulation  and  navigation  tasks) 

3.1  We  first  replicated  the  simple  version  of  the  MFT  algorithm  initially 
proposed  by  Fontanari  &  Perlovsky  (2005  KIMAS  Conference)  to 
model  category  formation  for  symbol  grounding. 

3.2  We  have  successfully  extended  the  MFT  to  deal  with  multi-feature 
object  classification,  which  is  typical  of  ours  cognitive  robotic  model  in 
which  we  have  multiple  features  to  presented  objects  and  action.  Results 
presented  in  the  ICANN06  paper. 

3.3  We  have  started  to  adapt  the  multi-feature  MFT  algorithm  to  model  the 
acquisition  of  words.  Initial  simulations  were  done  at  the  workshop  in 
Brazil  and  presented  at  ICANN2006  conferences. 
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4.  To  do  further  simulation  aiming  at  the  scaling 
up  of  the  agents’  lexicon  in  terms  of  high 
number  of  lexical  entries  and  types  of 
syntactic  categories 


4.1  initial  simulations  on  the  scaling  up  of  MFT  to  the  classification  of 
thousands  of  objects  and  models  have  demonstrated  the  feasibility  of  the 
algorithm  against  combinatorial  complexity. 

4.2  Results  on  the  scale  up  of  the  original  MFT  algorithm  are  available  on 
the  project  website  http://www.tech.plvm.ac.uk/vadim/Resultsl.htm 

4.3  Some  scale  up  simulations  on  the  multi-feature  MFT  algorithm  were 
included  in  the  paper  submitted  to  the  ICANN2006  conference. 

4.4  New  simulation  experiments  on  the  classification  and 
categorization  of  actions  and  the  building  of  sensorimotor  concept- 
models.  Use  of  a  set  of  112  actions  resembling  the  semaphore  flag 
signaling  system.  Results  demonstrate  that  the  MFT  algorithm  can 
easily  scale  up  to  the  use  of  numerous  actions/words  with  minimal 
parameter  tuning  (see  below  report  on  new  simulation  results) 

4.5  New  simulations  on  the  extension  of  the  MFT  algorithm  to  enable 
agents  to  learn  a  lexical  item  to  name  each  of  the  122  actions.  After 
performing  the  action,  the  agents  learn  to  describe  each  action  using 
three-letter  words  Consonant-Vowel-Consonant).  These 
simulations  show  that  agents  are  able  to  learn  a  lexicon  to  describe 
these  objects/actions  through  a  process  of  cultural  learning  (see 
below  report  on  new  simulation  results) 

4.6  New  simulations  on  acquisition  of  composite  actions.  First  the 
agent  learns  a  series  of  basic  actions  and  their  basic  names. 
Subsequently,  the  agent  is  able  to  successfully  combine  basic  actions 
into  higher  order  concepts  through  the  combination  of  the  action 
words,  as  in  a  simplified  grammar,  (see  below  report  on  new 
simulation  results) 

4.7  Plan  for  future  language-cognition  integration  experiments  with 
real  robot.  Cangelosi  and  Tikhanoff  are  currently  in  discussion  with 
Professor  Giulio  Sandini  (Italian  Institute  of  Technology,  Genoa)  to 
implement  some  of  the  above  experiments  with  one  of  the  humanoid 
robotic  platform  available  at  the  Italian  Institute  of  Technology. 
Tikhanoff  is  expected  to  spend  a  few  months  in  Genoa  to  work  on 
this  in  the  period  February-June  2006. 


ADDITIONAL  ACTIVITIES  BETWEEN  NOVEMBER  2005  -  MARCH  2006 

1.  Literature  review  for  PhD  and  project 

Tikhanoff.  under  the  supervision  of  Cangelosi,  started  in  November  2005  the  theoretical  preparatory  work  for  the  project.  His 
literature  review  has  focused  on  the  following  topics:  (a)  action  and  language  integration;  (b)  computational  models  of  the 
evolution  and  acquisition  of  language;  (c)  modelling  field  theory  for  cognitive  studies;  (d)  human-robot  interaction  and 
communication. 

2.  Organisation  of  Special  session  on  “Modelling  Language  Acquisition  and  Evolution”  at  the  IJCNN06  conference. 

Cangelosi  was  the  chair  of  the  Special  Session  on  “Modelling  Language  Acquisition  and  Evolution”  held  on  July  2006  at  the 
IJCNN2006  conference.  The  session  was  organized  in  collaboration  with  Dr.  Leonid  Perlovsky  and  Prof.  Jose’  Fernando 
Fontanari.  The  special  session  consisted  of  6  papers  refereed  through  the  standard  IJCNN  review  procedure. 

3.  Site  visit  by  Dr.  Paul  Losiewicz,  EOARD  London 

On  12-13  January  2006  we  had  a  site  visit  by  Dr.  Losiewicz  (see  enclosed  agenda).  During  this  visit  we  presented  the  early 
work  on  the  project.  Dr.  Losiewicz  also  met  with  various  research  groups  active  in  our  Faculty.  We  also  discussed  possible 
ideas  and  topics  for  follow-on  research,  which  will  be  discussed  soon  between  Dr.  Cangelosi  and  Dr.  Losiewicz. 

4.  Grant  meeting  and  workshop  in  Joao  Pessoa,  Brazil  (17-24  February,  2006) 

During  the  week  starting  on  February  17  we  had  a  very  useful  workshop  amongst  the  Plymouth  Team  (Cangelosi  and 
Tikhanoff),  Dr.  Perlovsky  (USAF  Hanscom,  MA)  and  Prof.  Fontanari  (University  of  San  Paulo  at  San  Carlos).  During  the 
workshop  we  discussed  and  developed  new  versions  of  the  MFT  algorithm  to  deal  with  multi-feature  object  representation  and 
to  extend  the  model  to  deal  with  word-word  and  word-object  association  learning  during  language  acquisition. 

5.  Webpage  for  grant 

Tikhanoff  has  designed  a  web  page  dedicated  to  the  project.  This  includes  an  overview  of  the  project,  results  of  simulations, 
and  publications  produced  by  the  team.  The  grant  webpage  is  available  at  the  following  website: 
http://www.tech.plym.ac.uk/vadim/ 


ADDITIONAL  ACTIVITIES  BETWEEN  APRIL  2006  -  OCTOBER  2006 


6.  Presentation  at  AFOSR  Workshop  At  Information  Fusion  Conference 

Professor  Cangelosi  presented  the  cognitive  robotic  model  at  the  AFOSR  WORKSHOP,  part  of  the  Information  Fusion 
Conference  in  Florence,  July  10-11,  2006.  The  participation  to  the  AFSOR  workshop  provided  an  excellent  opportunity  to 
meet  with  the  researchers  involved  in  information  fusion,  including  EOARD  contractors  from  European  universities,  AFRL 
researchers,  and  other  U.S.  academics  involved  in  AFRL  projects  and  to  get  an  up-to-date  overview  of  current  research 
projects  in  the  different  levels  of  information  fusion,  including  multisensor  fusion,  situation  awareness,  distributed  intelligent 
control. 

7.  Proposal  for  Special  Issue  in  the  journal  IEEE  Transactions  on  Neural  Networks 

Cangelosi  in  collaboration  with  Dr.  Perlovsky  and  other  colleagues  have  proposed  the  preparation  of  a  special  issue  in  the 
journal  IEEE  Transactions  on  Neural  Networks.  The  topic  of  the  special  issue  is  “Modelling  the  Mind  and  Brain”.  We  are 
currently  awaiting  for  the  outcome  of  our  proposal. 

8.  Grant  meetings  at  Evolang6  Conference  in  Rome. 

On  April  12-14,  Professor  Cangelosi  was  the  Conference  Chair  of  the  6th  International  Conference  on  the  Evolution  of 
Language  (Evolang6),  in  Rome.  Tikhanoff  was  also  involved  in  the  Organising  Committee.  During  the  conference  we  had 
some  grant  meetings  with  Perlovsky  and  Fontanari  to  discuss  ongoing  work  on  the  project  and  the  changed  to  MFT  algorithm 
for  the  scaling  up  experiments. 

9.  Featuring  of  research  in  New  Scientist  magazine 

Following  the  IJCNN  presentation  of  our  paper,  Cangelosi  was  approached  by  Tom  Simonite,  a  journalist  for  the  New 
Scientist  magazine,  for  an  article  on  our  research.  The  article  “Virtual  hots  teach  each  other  using  wordplay”  was  published  on 
August  2nd,  2006,  in  the  on-line  version  of  the  New  Scientist  magazine: 
http://www.newscientisttech.com/article.ns7kKdn9652 

10.  Interview  on  BBC  Radio 

Professor  Cangelosi  was  invited  to  give  a  live  interview  on  this  work  at  the  BBC  Radio  Devon  Breakfast  Show  on  August  31, 

2006.  The  interview  is  available  on  the  Pi’s  webpage: 

http://www.tech.plvm.ac.uk/soc/stafPangelo/ 

11.  Featuring  of  research  in  Professional  Engineering  magazine 

Last  August  Professor  Cangelosi  was  also  approached  by  Rebecca  Rushmer,  a  journalist  for  the  Professional  Engineering 
magazine,  for  an  article  on  this  research.  We  are  still  waiting  to  hear  about  the  publication  details  of  the  article. 

12.  Presentation  at  Science  Festival,  Genoa  (Italy) 

Cangelosi  has  been  invited  to  a  conference  on  “How  we  lean  to  speak:  Comparing  Babies  and  Machine”  at  the  Science  Festival 
in  Genoa,  Italy.  This  will  be  held  on  October  30,  2006.  Webpage  here: 
http://www .  festivalscienza ,  it/it/pro  gramma/ e  vento  ,php?id=  153 

13.  Interview  on  Italian  RAI  Radio 

Cangelosi  will  be  interview  on  the  research  on  robots  and  language  at  the  program  “Radio3  Scienza”  of  the  Italian  RAI  Radio 
Channel  3.  This  will  be  on  the  morning  of  October  30th,  2006. 

14.  Participation  to  Summer  School  in  Cognitive  Robotics 

Tikhanoff  was  selected  and  attended  the  Veni  Vidi  Vici’06  Robocub  summer  school  in  July  2006  (Ventimiglia,  Italy). 
http://eris.liralab.it/summerschool/ 

During  the  school  he  applied  a  version  of  the  multi-dimensional  MFT  algorithm  on  a  23  degree  of  freedom  humanoid  robot  to 
successfully  achieve  categorization  of  actions. 


SCHEDULE  OF  REPORTS/DELIVERIES 

(in  bold  deliverable  between  April  and  October  2006) 


Original  deliverables 

Actual  deliverables 

INTERMEDIATE  REPORT:  March  28,  2006 

Report  on  the  design  of  model  and  preliminary 
testing  and  Conference  paper  submission; 

Prototype  of  simulator  for  robotic  experiments  on 
grounding  communication  in  sensorimotor  and 

See  Intermediate  report  submitted  to  EOARD  in  March  2006 

cognitive  tasks. _ 

1.  Presentation  of  IJCNN  paper  at  IEEE  World  Congress  on 
Computational  Intelligence,  Vancouver,  on  16-21  July  2006 

2.  Organization  of  special  session  at  IJCNN 


FINAL  REPORT:  October  31,  2006: 


July  30, 2006:  Presentation  of  work  (e.g.  IEEE 
World  Congress  on  Computational 
Intelligence,  Vancouver) 


Final  report  on  simulation  experiments  and 
preparation  of  paper  for  neural  network 
and/or  robotics  journal  (e.g.  Neural 
Computation  or  IEEE  Trans.  Neural 
Networks); 


A  new  delivery  is  the  paper  that  will  be  presented  at  the  IEEE 
KIMAS  Conference  in  Boston  (April  2007). 

The  report  enclosed  at  the  end  of  this  document  contains  a 
detailed  description  of  the  new  experiment  on  scaling  up  of 
lexicon  and  action  repertoire.  Some  of  the  results  will  be  included 
in  the  KIMAS  presentation.  The  report,  together  with  the  other 
simulation  published  in  the  ICANN  and  IJCNN  papers,  will 
constitute  the  core  of  the  journal  paper  that  is  going  to  be 
prepared  for  submission  to  IEEE  Transactions  on  Neural 
Networks  (possibly  as  part  of  the  special  issue  on  Mind  and  Brain 
that  we  have  proposed  to  the  editor) 


Final  release  of  robotic  simulator. 


The  simulator  for  the  robotic  experiments  and  the  MET  language 
and  action  training  is  available  in  the  grant  webpage: 
http://www.tech.plym.ac.uk/vadim/Images/Results/ODE  MFT.zip 
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Tikhanoff  V.,  Cangelosi  A.,  Fontanari  J.F.  &  Perlovsky  L.I.  (2006).  Scaling  up  of  action  repertoire  in  linguistic  cognitive 

agents.  KIMAS07:  IEEE  International  Conference  on  Integration  of  Knowledge  Intensive  Multi-Agent  Systems,  Boston, 
April  2007 
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Experiments  on  action-language  integration.  IEEE  Transactions  in  Neural  Networks. 


Report  on  New  Simulations  on  the  Scaling  up  of  lexicon  and  action  repertoire  in 

cognitive  agents 


Recent  research  in  autonomous  cognitive  systems  has  focused  on  the  close  integration  (grounding)  of  language  with  perception 
and  other  cognitive  capabilities.  In  this  paper  we  propose  the  utilization  of  the  Modeling  Field  Theory  (MFT)  to  deal  with  the 
combinatorial  complexity  problem  of  language  modeling.  MFT  aims  at  overcoming  such  limitations  by  dynamic  logic  learning 
of  lower-level  signals  (e.g.,  inputs,  bottom-up  signals)  with  hierarchies  of  higher-level  concept-models  (e.g.  internal 
representations,  categories/concepts,  top-down  signals).  This  is  the  case  of  language,  which  is  characterized  by  the  hierarchical 
organization  of  underlying  cognitive  models. 

In  this  paper  we  present  an  integration  of  the  Modeling  Field  Theoiy  algorithm  for  the  classification  of  objects  with  a  model  of 
the  acquisition  of  language  in  cognitive  robotics.  In  new  simulations  we  have  applied  and  extended  our  previous  modified 
version  of  the  MFT  algorithm  to  deal  with  the  scaling  up  of  the  robotic  agent’s  action  repertoire.  Simulations  are  divided  into 
two  stages.  First  agents  learn  to  classify  112  different  actions  inspired  by  an  alphabet  system  (the  semaphore  flag  signalling 
system).  In  the  second  stage,  agents  also  learn  a  lexical  item  to  name  each  action.  At  this  stage  the  agents  will  start  to  describe 
the  action  as  a  “word”  comprised  of  three  letters  (consonant  -  vowel  -  consonant).  The  outcome  of  the  simulations  is  that:  (i) 
agents  are  able  to  acquire  a  complex  set  of  actions  by  building  sensorimotor  concept-models;  (ii)  agents  are  able  to  learn  a 
lexicon  to  describe  these  objects/actions  through  a  process  of  cultural  learning,  (iii)  agents  learn  actions  as  basic  gestures  in 
order  to  generate  composite  actions. 


Introduction 


Modeling  Field  Theory 

Modeling  Field  Theory  is  based  on  the  principle  of  associating  lower-level  signals  (e.g.,  inputs,  bottom-up  signals)  with 
higher-level  concept-models  (e.g.  internal  representations,  categories/concepts,  top-down  signals)  avoiding  the  combinatorial 
complexity  inherent  to  such  a  task.  This  is  achieved  by  using  measures  of  similarity  between  concept-models  and  input  signals 
together  with  a  new  type  of  logic,  so-called  dynamic  logic.  MFT  may  be  viewed  as  an  unsupervised  learning  algorithm 
whereby  a  series  of  concept-models  adapt  to  the  features  of  the  input  stimuli  via  gradual  adjustment  dependent  on  the  fuzzy 
similarity  measures. 

Modeling  Field  Theory  combines  neural  architecture  with  models  of  objects.  For  feature-based  object  classification  considered 


here,  each  input  neuron 


;=1,...,A 


encodes  feature  values  '  (potentially  a  vector  of  several  features);  each  neuron  1  may 

r  fmm  i  rro  1  /a  n  t  prtnfpvt  r>  1 1 1  nr  nmop  \\/ p  fprm  flip  cpf  ^  a  n  innut  npnro 


contain  a  signal  from  a  real  object  or  from  irrelevant  context,  clutter,  or  noise.  We  term  the  set  ■  ’  an  input  neural 

field:  it  is  a  set  of  bottom-up  input  signals.  Top-down,  or  priming  signal-fields  to  these  neurons  are  generated  by 

models,  ^ k )  where  we  enumerate  models  by  index pac|1  model  is  characterized  by  its  parameters  * ,  which 
may  also  be  a  vector  of  several  features.  In  this  contribution  we  will  consider  the  simplest  possible  case,  in  which  parameters 

M  (S  )  =  S 

model  represent  feature  values  of  object,  k‘  k  Interaction  between  bnttnm-un  and  tnn-dnwn  siunaR  is  determined  b\ 


.  Interaction  between  bottom-up  and  top-down  signals  is  determined  by 
vs.  We  introduce  an  arbitrary  similarity  measure  ^  ^  between 


neural  weights  associating  signals  and  models  as  follows.  We  introduce  an  arbitrary  similarity  measure 
O  S 

bottom-up  signals  f  and  top-down  signals  1  [see  equation  (2)],  and  define  the  neural  weights  by 


f(k\i)  =  l(i\k)/Zl(i\k') 


These  weights  are  functions  of  the  model  parameters  ,  which  in  turn  are  dynamically  adjusted  so  as  to  maximize  the  overall 
similarity  between  object  and  models.  This  formulation  sets  MFT  apart  from  many  other  neural  networks. 


The  model 

We  consider  the  problem  of  categorizing  N  objects  i  =  1,  •  -  - ,  N,  each  of  which  characterized  by  d  features  e  =  1,  •••,£/.  These 
features  are  represented  by  real  numbers  Oie  e  (0,1)  -  the  input  signals  -  as  described  before.  Accordingly,  we  assume  that  there 
are  M  t/-dimensional  concept-models  k  =  described  by  real-valued  fields  Ske ,  with  e  =  l as  before,  that  should 


match  the  object  features  Ou  .  Since  each  feature  represents  a  different  property  of  the  object  as,  for  instance,  color,  smell, 
texture,  height,  etc.  and  each  concept-model  component  is  associated  to  a  sensor  sensitive  to  only  one  of  those  properties,  we 
must,  of  course,  seek  for  matches  between  the  same  component  of  objects  and  concept-models.  Hence  it  is  natural  to  define  the 
following  partial  similarity  measure  between  object  i  and  concept  k 

l(i  |  k)  =  n^rexp  [-  (.S'fc  -Ole)2 / 2  a l  ]  (2)  where,  at  this  stage,  the  fuzziness  crfe  is 

e=l 

a  parameter  given  a  priori.  The  goal  is  to  find  an  assignment  between  models  and  objects  such  that  the  global  similarity 
Z  =  Xlog£/(/|k)  (3) 

i  k 

is  maximized.  This  maximization  can  be  achieved  using  the  MFT  mechanism  of  concept  formation  which  is  based  on  the 
following  dynamics  for  the  modeling  field  components 

dStJdt  =  f(k  |  /)[Slog/(/ 1  k)/dSh\,  (4)  which,  using  the  similarity  (1),  becomes 

dSjdt  =  -Yjf{k\i){Ske-Oie)lale.  (5) 

Here  the  fuzzy  association  variables  /(k|/)are  the  neural  weights  defined  in  equation  (1)  and  give  a  measure  of  the 
correspondence  between  object  i  and  concept  k  relative  to  all  other  concepts  k  These  fuzzy  associations  are  responsible  for 
the  coupling  of  the  equations  for  the  different  modeling  fields  and,  even  more  importantly  for  our  purposes,  for  the  coupling  of 
the  distinct  components  of  a  same  field.  In  this  sense,  the  categorization  of  multi-dimensional  objects  is  not  a  straightfoiward 
extension  of  the  one-dimensional  case  because  new  dimensions  should  be  associated  with  the  appropriate  models.  This 
nontrivial  interplay  between  the  field  components  will  become  clearer  in  the  discussion  of  the  simulation  results. 

It  can  be  shown  that  the  dynamics  (4)  always  converges  to  a  (possibly  local)  maximum  of  the  similarity  L  [14],  but  by  properly 
adjusting  the  fuzziness  <rte  the  global  maximum  often  can  be  attained.  A  salient  feature  of  dynamic  logic  is  a  match  between 
parameter  uncertainty  and  fuzziness  of  similarity.  In  what  follows  we  decrease  the  fuzziness  during  the  time  evolution  of  the 
modeling  fields  according  to  the  following  prescription 

K(0  =  <y]  exp(-  at)+a2h  (6) 

with  a  =  5  x  1IT4 ,  cra  =  1  and  ah  -  0.03  .  Unless  stated  otherwise,  these  are  the  parameters  we  will  use  in  the  forthcoming 
analysis. 


Simulations 

In  this  section  we  will  report  results  from  3  simulations.  Initially  they  will  be  aimed  at  a  simple  scaling  up  of  the  agent’s  action 
repertoire  using  multi-dimension  features.  In  the  second  simulation  we  will  demonstrate  the  correct  classification  of  the  input 
object  though  the  dynamic  introduction  of  the  lexicon  feature.  The  third  simulation  will  concentrate  on  breaking  down  the 
actions  into  basic  gestures  in  order  to  generate  composite  actions. 

To  facilitate  the  presentation  of  the  results,  we  will  interpret  both  the  object  feature  values  and  the  modeling  fields  as  d  - 
dimensional  vectors  and  follow  the  time  evolution  of  the  corresponding  vector  length 

st=^(sj  /d, 

which  should  then  match  the  object  length  Oj  =  /d- 

Simulation  I:  Classification  and  categorization  of  actions  /  building  sensorimotor  concept-models 

Let’s  first  consider  having  112  different  actions,  some  inspired  by  an  alphabet  system  (the  semaphore  flag  signalling  system) 
see  figure  2.  We  have  collected  data  on  the  posture  of  robots  using  6  features.  The  object  input  data  consist  of  the  6  angles  of 
each,  left  arm  and  right  arm  joints.  (Shoulder,  upper  arm  and  elbow)  The  agents  first  have  to  learn  to  classify  these  actions;  at 
this  stage  we  are  using  a  multi-dimensional  MFT  algorithm  with  1 12  fields  randomly  initialized.  Figure  1  shows  that  the  model 
is  able  to  correctly  identify  the  different  actions.  Although  the  simulation  initially  dealt  with  1 12  actions  the  MFT  algorithm 
was  able  to  categorize  to  approximately  95%  successful  matching.  Therefore  there  was  a  slight  reduction  in  the  number  of 
completed  actions.  Figure  3  shows  our  system  consisting  of  two  simulated  agents  (teacher  and  learner)  embedded  within  a 
virtual  simulated  environment  (Using  Open  Dynamic  Engine). 


Classification  and  categorization  of  actions 


t/50 


Figure  1:  Time  evolution  of  the  fields  with  6  features  being  used  as  input:  1 12  different  actions. 
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Figure  2:  Few  examples  of  type  of  behaviour  used  for  the  classification  and  categorization  of  actions,  (here  the  semaphore  alphabet) 


 Simulation  test  environment  v0.02  «  □ 

n  Simulation  test  environment  v0.02  |-J  □ 

File  Simulation  Help 

File  Simulation  Help 

A '  z '/ '  iz  t :  V.YX  X 

^  ■  * 

Figure  3:  Teacher  and  learner  before  (left)  and  after  (right)  the  action  is  learnt. 


Simulation  II:  Incremental  Feature  -  lexicon  acquisition 

In  the  first  simulation  we  have  proposed  the  use  of  the  multi-dimensional  MFT  in  order  to  categorize  1 12  different  actions.  At 
this  stage  we  wanted  to  explore  the  integration  of  language  and  cognition  in  cognitive  robotic  studies.  Here  we  extend  the 
multi-dimensional  MFT  algorithm,  used  in  Simulation  1,  to  enable  the  agents  to  learn  a  lexical  item  to  name  each  previous 
action.  After  performing  the  action,  the  agents  will  start  to  describe  this  action  as  three  letter  word,  (consonant  -  vowel  - 
consonant)  see  example  table  1 .  Each  word  is  unique  to  the  action  performed.  This  phonetic  feature  is  dynamically  added 
immediately  after  the  action.  At  timestep  12500,  (half  of  the  training  time)  both  features  are  considered  when  computing  the 
fuzzy  similarities.  From  timestep  12500,  the  dynamics  of  the  o2  fuzziness  value  is  initialized,  following  equation  (6),  whilst  ol 
continues  its  decrease  pattern  started  at  timestep  0.  Results  in  figure  4  show  that  the  model  is  able  to  categorize  an  action  and 
assign  a  ‘word’  to  this  action. 


:  Incremental  Feature  -  lexicon  acquisition 
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Figure  4:  Time 

evolution  of  the  fields  using  as  input  the  action  and  phonetic  feature:  1 12  different  actions  +112  words 
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Actions  Words  Actions  Words  Actions  Words  Actions  Words 

1 

XUS 

29 

WED 

57 

LAC 

85 

KUN 

2 

GAP 

30 

JIL 

58 

BIF 

86 

BAM 

3 

HES 

31 

SAK 

59 

FUF 

87 

TOC 

4 

LET 

32 

BAR 

60 

HOR 

88 

NOK 

5 

PAQ 

33 

JOC 

61 

QEB 

89 

TIL 

6 

PAL 

34 

FUV 

62 

WAH 

90 

HAW 

7 

VEH 

35 

MEL 

63 

SUD 

91 

JOJ 

8 

WIG 

36 

JAX 

64 

KAW 

92 

WOT 

9 

FUH 

37 

JAS 

65 

SAV 

93 

REW 

10 

TEP 

38 

VUS 

66 

QEQ 

94 

FUN 

11 

LUT 

39 

KEK 

67 

GES 

95 

CIH 

12 

COF 

40 

MIQ 

68 

WEJ 

96 

KUM 

13 

QUT 

41 

PAJ 

69 

KEJ 

97 

VEF 

14 

TUW 

42 

DOJ 

70 

LUC 

98 

NUH 

15 

REV 

43 

HER 

71 

KAC 

99 

GAC 

16 

QER 

44 

SAR 

72 

GUO 

100 

KAC 

17 

BIJ 

45 

TOD 

73 

PUB 

101 

GUG 

18 

QUR 

46 

COC 

74 

RIV 

102 

CEJ 

19 

QUX 

47 

KAH 

75 

DEM 

103 

FUG 

20 

XAM 

48 

BIG 

76 

BER 

104 

KOG 

21 

BOX 

49 

QUW 

77 

WIP 

105 

VEX 

22 

QIH 

50 

PIM 

78 

PUP 

106 

WAK 

23 

PUV 

51 

DET 

79 

SIK 

107 

WOW 

24 

FAC 

52 

FUJ 

80 

XIT 

108 

BER 

25 

SIL 

53 

MIK 

81 

VEN 

109 

CEB 

26 

TAX 

54 

MEG 

82 

FIP 

110 

TOK 

27 

HUT 

55 

RAQ 

83 

SIC 

111 

TES 

28 

DOK 

56 

MAK 

84 

wuw 

112 

LOP 

Table  1:  Table  containing  examples  of  words  generated  by  the  agents  for  a  specific  action 
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Figure  5:  Teacher  and  learner  before  action  is  learnt  and  after  with  the  addition  of  the  ‘word’.  For  visualization  purposes,  the  word  is  added 
on  the  image. 


Simulation  III:  Progressive  learning  of  basic  gestures  into  composite  actions 

Previous  simulations  consisted  of  learning  actions  or  a  combination  of  actions  and  words.  In  this  final  simulation  we  are  taking 
a  step  backwards  in  the  categorization  of  actions.  We  are  breaking  down  the  action  into  basic  gestures.  Before  learning  a 
completed  action  we  are  interested  in  the  systematic  breakdown  of  actions  into  individual  gestures,  that  is  to  say  for  example  a 


two-handed  action  would  be  broken  down  into  two  single  handed-actions  and  analyzed  as  individual  steps  in  the  process  of  a 
compound  action.  As  an  extension  to  the  previous  simulations,  each  feature  is  added  dynamically.  Firstly  the  simulation  starts 
with  the  left-handed  action.  At  timestep  10000  (l/3rd  of  the  simulation)  we  consider  the  right-handed  action,  using  the  same 
dynamics  of  the  fuzziness  values  as  for  simulation  2  and  finally  at  timestep  20000  we  consider  the  phonetic  feature.  Figure  6 
shows  that  the  model  is  able  to  dynamically  adapt  to  compound  action  associated  with  the  word  generation. 


Progressive  learning  of  basic  gestures  into  composite  actions 


Discussion  and  conclusion 

In  this  paper  we  present  an  integration  of  the  Modeling  Field  Theory  algorithm  for  the  classification  of  objects  with  a  model  of 
the  acquisition  of  language  in  cognitive  robotics.  In  new  simulations  we  have  applied  and  extended  our  previous  modified 
version  of  the  MFT  algorithm  to  deal  with  the  scaling  up  of  the  robotic  agent’s  action  repertoire.  The  various  simulations 
showed  that  (i)  agents  are  able  to  acquire  a  complex  set  of  actions  by  building  sensorimotor  concept-models;  (ii)  agents  are 
able  to  learn  a  lexicon  to  describe  these  objects/actions  through  a  process  of  cultural  learning,  (iii)  agents  learn  actions  as  basic 
gestures  in  order  to  generate  composite  actions. 


